class: center, middle, inverse, title-slide # Lecture 6 ## Model Evaluation ### Psych 10 C ### University of California, Irvine ### 04/11/2022 --- ## Summary - Last week we started with two different hypotheses about the relation between lung capacity and smoking status. -- - The Null model stated that there where no differences in lung capacity as a function of smoking status, the model was formalized as: `$$y_{ij}\sim\text{Normal}(\mu,\sigma^2)$$` -- - We found two estimators for the parameters in the model, `\(\hat{\mu}\)` which is the average of the participants lung capacity regardless of smoking status. -- - And `\(\hat{\sigma}^2_0\)` which is the error or variability of our observations when we use `\(\hat{\mu}\)` as a prediction of each observation. -- - Finally we said that we would be interested on the Sum of Squared Errors of the Null model which is defined as: `$$SSE_0 = \sum_j \sum_i \left(y_{ij}-\hat{\mu}\right)^2$$` --- ## Summary - Our second hypothesis was the Effects Model, which assumes that there is a difference in lung capacity as a function of smoking status. This model is formalized as: -- `$$y_{ij}\sim\text{Normal}(\mu_j,\sigma_e^2)$$` -- - The estimator of our parameter `\(\mu_j\)` (one for each group `\(\hat{\mu}_j\)`) was equal to the average of each group (taken independently). -- - The estimator for `\(\sigma_e^2\)` was equal to the error of the model, which is the average squared difference between each observation and the model's prediction for that observation `\(\hat{\mu}_j\)`. -- - Finally, we mention that we will also be interested in the Sum of Squared Errors of the Effects Model, which is defined as: `$$SSE_e = \sum_j \sum_i \left(y_{ij}-\hat{\mu}_j\right)^2$$` --- ## Adding predictions - The firs thing that we want to do is add our predictions and the squared error of each observation to our data. -- ```r # total sample size n_total <- nrow(smokers) # get the predictions of effects model (\hat{\mu}_j) eff_pred <- smokers %>% group_by(smoke_status) %>% summarise("prediction" = mean(lung_capacity)) # add predictions to data smokers <- smokers %>% mutate("pred_null" = rep(x = mean(lung_capacity), times = n_total), "pred_eff" = ifelse(test = smoke_status == "smoker", yes = eff_pred$prediction[2], no = eff_pred$prediction[1])) ``` --- ## Adding errors - Now to add the squared errors we can use the difference between prediction and observation squared: ```r smokers <- smokers %>% mutate("error_null" = (lung_capacity - pred_null)^2, "error_eff" = (lung_capacity - pred_eff)^2) ``` -- - Now our data file has the relevant variables for each observation:
--- ## Sum of Squared Errors - Using the updated table it's easy to get the values of the SSE and the estimators `\(\hat{\sigma}_0^2\)` and `\(\hat{\sigma}_e^2\)`. ```r # the sse of the null model is: sse_0 <- sum(smokers$error_null) # the sse for the effects model is: sse_e <- sum(smokers$error_eff) # mean sse null model sigma_0 <- 1/n_total * sse_0 # mean sse effects model sigma_e <- 1/n_total * sse_e ``` -- Their values are: .pull-left[ Null Model `\(SSE_0\)` = 252.36 `\(\hat{\sigma}_0^2\)` = 31.54 ] .pull-right[ Effects Model `\(SSE_e\)` = 46.31 `\(\hat{\sigma}_e^2\)` = 5.79 ] --- class: inverse, center, middle # Model Evaluation ## `\(R^2\)` --- ## Model Evaluation: `\(R^2\)` - `\(R^2\)` is a method that allows us to "measure" how good a general model is in comparison to a **nested** model. -- - In our problem, the Null model is nested on the Effects model, but what does it mean for a model to be nested? -- - We say that one model is nested on another when the values that the parameters of the nested model are a special case of the second. -- - For example, in our effects model there is nothing that makes it impossible for `\(\mu_1\)` to be equal to `\(\mu_2\)`, which would be the same as saying that the expectation of each group was the same... -- - But that was the assumption of the **Null Model**! -- - This means that the Null model is nested in the Effects Model, in other words, the **Null Model** is a special case of the **Effects Model**. --- ## Model Evaluation: `\(R^2\)` - What does it mean for our results for the Null Model to be nested on the Effects Model? -- - Well this means that, in terms of error or variability, the Effects Model will always have a lower than or equal error than the Null Model. -- - In other words, the worse that the Effects Model can do is to be equal to the Null Model, everything else will make the Effects Model better. -- - We can express this new information formally using the following equation: `$$SSE_0 = SSE_a + SSE_e$$` -- - Where the new variable `\(SSE_a\)` represents the error or variation that is reduced or accounted for when we use the Effects Model In comparison to the Null. -- - Given that we know that `\(SSE_e\)` will always be **equal or slower** than `\(SSE_0\)` we can express the error accounted for by the Effects Model as: `$$SSE_a = SSE_0 - SSE_e$$` --- ## Model Evaluation: `\(R^2\)` - `\(R^2\)` can be interpreted as the proportion of error accounted for by the effects model out of the total error. -- - The total error is the error of the Null model, and the error accounted for is `\(SSE_a\)`. Therefore, we can express the proportion of accounted error `\((R^2)\)` as: `$$R^2 = \frac{SSE_a}{SSE_0} = \frac{SSE_0 - SSE_e}{SSE_0}$$` -- - By definition we have that: `$$0 \leq R^2 \leq 1$$` --- ## Model Evaluation: `\(R^2\)` - When `\(R^2 = 0\)` there is no difference in the error (variance) of the models. -- - When `\(R^2 = 1\)` it means that the error (variance) of the Effects model is 0 (which will can't happen unless that our observations have no variability). -- - This means that the closer the value of `\(R^2\)` is to 1 the better the effects model is! -- - Whenever you see a Journal paper that uses a linear model it is likely that you will encounter a value of `\(R^2\)` defined as the proportion of error accounted for by the Effects model (or linear model). -- - Let's calculate the proportion of error accounted for in our smokers example. --- ## `\(R^2\)`: Smokers Experiment - Using the Sum of Squared Errors that we calculated before we can obtain the proportion of error accounted for by the effects model. -- - Go back to slide 6 and using the values there calculate the proportion of error accounted for by the effects model in the smokers data set. -- <br> .can-edit.key-likes[ `\(R^2 =\)` ] --- ## `\(R^2\)` - `\(R^2\)` will give us a first look at how the Effects model is doing in a given situation. -- - However, there is a problem with `\(R^2\)` that can't be avoided... -- - Remember that we said that the error (variance) of the Effects model will always be equal or lower than that of the Null model. -- - This is because using two distributions instead of one gives us more flexibility. -- - In other words, we can always do better using two normal distributions than usin a single one as in the Null model. -- -